Information drives business. A hardware or software failure affecting a data center can cause days or even weeks of unplanned downtime and data loss that could threaten an organization's productivity. For businesses that increasingly depend on data and information for their day-to-day operations, this unplanned downtime can also hurt their reputations and bottom lines. Businesses are becoming increasingly aware of these costs and are taking measures to plan for and recover from hardware and software failures.
Many complex business applications are run not on a single computer system, but in a distributed system in which multiple computer systems, referred to as nodes, each contribute processing resources and perform different tasks. In such an environment, disruption due to hardware and software failures can be lessened or prevented using a strategy known as clustering. In a clustered environment, computer systems and storage devices are interconnected, typically at high speeds within a local data center, for the purpose of improving reliability, availability, serviceability, and/or performance via load balancing. Redundant interconnections between the computer systems are typically included as well, and the collection of computer systems, storage devices, and redundant interconnections is referred to herein as a cluster. In some implementations, the cluster appears to users as a single highly available system. Different types of clusters may be established to perform independent tasks, to manage diverse hardware architectures performing similar tasks, or when local and backup computer systems are far apart physically.
In some clustering environments, only one of the computer systems in the cluster provides processing resources with respect to a particular software application. In other clustering environments, processing for a single software application is distributed among nodes in the cluster to balance the processing load.
Within a single computer system, multiple threads executing a given software application may access and/or update the same data or resources. The term ‘thread’ is used to describe the context in which a computer program is being executed. This context includes the program code, the data for execution of the program code, a stack, a program counter indicating a memory location from which the next instruction will come, and state information. Coordination is necessary to ensure that one thread does not read shared data or access a shared resource at the same time that another thread is updating that data or resource, thereby possibly resulting in an inconsistency or conflict depending upon the timing of the two operations. In clustering environments where processing for a given software application is “load balanced,” threads that share data or resources can be running on different nodes within a cluster.
Coordination between threads accessing shared data or resources is often implemented using locks. Typically, a lock is software that protects a piece of shared data or a resource; for example, in a file system, a lock can protect a file or a disk block. In a distributed system, a lock can also protect shared “state” information distributed in memories of each node in the system, such as the online or offline status of a given software application. All shared data is protected by a lock, and locks are typically managed by a lock manager, which often provides an interface to be used by other application programs.
A lock is requested before the calling application program can access data protected by the lock. A calling application program can typically request an “exclusive” lock to write or update data protected by the lock or a “shared” lock to read data protected by the lock. If the calling application program is granted an exclusive lock, then the lock manager guarantees that the calling program is the only thread holding the lock. If the calling program is granted a shared lock, then other threads may also be holding shared locks on the data, but no other thread can hold an exclusive lock on that data.
The lock manager cannot always grant a lock request immediately. Consider an example in which one thread has an exclusive lock on a given set of data, and a second thread requests shared access to the given set of data. The second thread's request cannot be granted until the first thread has released the exclusive lock on the given set of data.
A lock can be placed on data that are stored on a shared disk. Locks can also be placed on shared data stored in memory for each node, where the data must be consistent for all nodes in a cluster. For example, nodes in a cluster can share information indicating that a file system is mounted. A lock can be placed on the shared state information when the state of the file system changes from mounted to not mounted, or vice versa.
In situations in which a process running in the user level of a node needs access to either system or cluster-wide resources protected by locks, such locks are generally implemented in the kernel of a node. To allow the user process to use the kernel-level locks, a user-kernel interface is provided. An issue with such a user-kernel interface is if the user-level process is abnormally terminated before releasing the lock, the lock can remain orphaned in a held state in the kernel. What is therefore needed is a solution that addresses a situation in which one or more locks are orphaned by the abnormal termination of a process holding the lock(s).